Skip to content

chore: Sample code to prototype using MPU to implement write methods that currently use resumable uploads#3448

Closed
ShreyasSinha wants to merge 7 commits intomainfrom
mpu-feature-13
Closed

chore: Sample code to prototype using MPU to implement write methods that currently use resumable uploads#3448
ShreyasSinha wants to merge 7 commits intomainfrom
mpu-feature-13

Conversation

@ShreyasSinha
Copy link
Copy Markdown
Collaborator

@ShreyasSinha ShreyasSinha commented Dec 22, 2025

Implementing a WriteChannel for uploading objects to Google Cloud Storage using the multipart upload API.
We breaks the data into 5MB chunks and uploads them in parallel using a fixed-size thread pool.
When the channel is closed, it sends a request to GCS to complete the multipart upload, assembling the parts into the final object.

Sample Benchmark: (Writing 5GB via cloudtop)
Using JSON writer with XML MPU API (Pool_SIZE = 16)
--- Aggregate Metrics ---
Total time: 21.179 s
File size: 5000.000 MB
Throughput: 236.082 MB/s

Using gRPC (existing)
--- Aggregate Metrics ---
Total time: 52.980 s
File size: 5000.000 MB
Throughput: 94.376 MB/s

Code used for benchmarking: https://github.com/ShreyasSinha/gcs-xml-mpu-benchmark/blob/main/src/main/java/org/example/ResumableUploadBenchmark.java

Test failures expected as its a prototype code.

@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: storage Issues related to the googleapis/java-storage API. labels Dec 22, 2025
@ShreyasSinha ShreyasSinha changed the title Sample code to prototype using MPU to implement write methods that currently use resumable uploads chore: Sample code to prototype using MPU to implement write methods that currently use resumable uploads Dec 22, 2025
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Dec 22, 2025
@ShreyasSinha ShreyasSinha added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Jan 16, 2026
@chingor13 chingor13 closed this Mar 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the googleapis/java-storage API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. size: l Pull request size is large.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants